Controlling a Universal Robot Handsfree — Quickstart

Go to the profile of Oz Ramos
Oz Ramos
Using JavaScript and a standard webcam you can “puppeteer” a robot!

I recently had an opportunity to hang out at the STUDIO at Carnegie Mellon University. During my stay I explored ways to interact with devices hands-free through a standard webcam using a creative coding tool I’ve been working on called Handsfree.js.

My goal with this project was to do a short performance piece puppeteering a robotic head with my own, and in this guide I’ll show you how you can too!

Here’s what I see and what the robot sees as I move my head

Before we begin though you’ll need to have the following ready:

Table of Contents

Here’s what we need to do to control the robot:

Starting the Robot and RoboDK

Starting the Robot

The first step is to power the robot on and initialize it (see my notes if you need a refresher). Next, connect to the robot and confirm that it has an IP address. If it doesn’t have one, run ipconfig (Windows) or ifconfig (Mac/Linux) from your computers terminal and use the values that come up under Ethernet.

On Windows, you should get an auto-assigned IP starting with 169.254.x.x when connecting to the robot. The preferred DNS server is my computers local IP

Starting RoboDK

Once you’ve configured the network on the robot it’s time to connect to it with your computer through RoboDK. To do so:

  • Open RoboDK
  • File > Open
  • Open the file named “UR5”

For more details, see: https://github.com/labofoz/handsfree-ur5/wiki/Setup---RoboDK. This is what you should see:

What you see when you load the included UR5.robot workspace. In the next section you’ll be clicking on A to connect to the robot, and B to load the server

Connecting and “Setting” the Robot

Connecting to the Robot

Once you have the UR5 workspace open in RoboDK, it’s time to connect! Right click on the UR5 to open the connection view:

Clicking this will bring up:
Make sure to input your robot’s IP and use port 30001

From the Connection to UR5 view above, input in the robot’s IP. It’s especially important to use port 30001 as this is the port the robot uses to listen for commands. If for some reason 30001 doesn’t work, then 30002 is the fallback. We won’t be connecting just yet, but click the Ping button to make sure we’re connected.

This screen has a few helpful buttons: Move Joints moves the robot to the exact pose seen in RoboDK. The Get Position button does the reverse, setting the simulator to match the physical robots pose. But before we start moving the physical robot, let’s finish “setting”.

“Setting” the Robot

Because we are puppeteering the robot so that the tool mimics our head’s motion, it’s important to “set” the robot’s origin in such a way that we have full range of motion no matter how we move ourselves.

The default pose on the UR5 robot is great for this:

The default UR5 pose in RoboDK is set so the tool won’t collide with itself. Consider this vs…
…a poorly set robot that may result in a collision the moment you turn on the webcam!

For the purpose of this guide, make sure to load a fresh copy of UR5 so that it’s pose looks like the first GIF in this section. Then, click the Move Joints button to have the robot move to match the simulator’s pose.

Your robot should now be “set”.

Load a WebSocket Server Onto the Robot

Almost there! The next step is to load a WebSocket server onto the robot, which will let us connect to the robot using JavaScript — drastically reducing the barrier to entry.

You’ll need Python 3.6+ installed, along with the following dependencies:

  • pip install websockets
  • pip install asyncio

Then:

A) Click Add Python Program
B) Right Click the added Python Program
C) Click Edit Python Script

When you click Edit Python Script , an editor will open up. Paste in the microserver here:

Send Commands to the Server

Start the Server

Finally, you’ll want to run the script to start the server. It’s a good idea to always test scripts inside the simulator before trying it on the robot. To do so, either double click the Python Program or Right Click > Run on Robot :

Run on robot runs the script inside the simulator; do that first! Once you’re ready to test on the physical robot click Upload to robot . To stop a running script, double click on the icon again or Right Click > Stop Python Script :

Next, visit the Handsfree Dashboard. I’ll be covering the dashboard later (and you can view the source on Glitch):

A) Enter the WebSocket URI to the robot’s server, which is ws:IP_ADDRESS:PORT_NUMBER . You can see the IP/Port used in the final few lines of the server’s code. If you copy/pasted then it should just be ws:localhost:1337

B) Click this to connect to the socket

C) This is the preview, which is toggle-able with D) if you need to improve performance

E) Toggle the webcam on/off to enable tracking

F) Check out the source code or grab it from the client/ folder in the repo: https://github.com/labofoz/handsfree-ur5

Fork and Explore!

Assuming I didn’t miss any steps, you should now be able to start the server and connect to it through the Handsfree Dashboard! Here are some closing thoughts:

Adding Functionality

To add or remove functionality, edit the server code. For instance, to rotate along a single axis, change the server’s code from:

target_i = target_ref \
                    * rotx(data['face']['rotationX'] * -1.5) \
                    * roty(data['face']['rotationY'] * -1.5) \
                    * rotz(data['face']['rotationZ'] * 1.5)

to

target_i = target_ref * rotx(data['face']['rotationX'] * -1.5)

Sending More Data

To manage what data is sent to the server, edit the Handsfree Dashboard. For example, to send translation information, change client.js from:

/**
  * Sends a message to the socket
  */
sendMessage: throttle(function (a) {
  lastPose && socketConnected && socket && socket.send(
  JSON.stringify({
    handsfree: true,
    action: 'updateCursor',
    cursor: lastPose.cursor,
    face: {
      rotationX: lastPose.face.rotationX,
      rotationY: lastPose.face.rotationY,
      rotationZ: lastPose.face.rotationZ
    }
  }))
}, 400)

to

/**
  * Sends a message to the socket
  */
sendMessage: throttle(function (a) {
  lastPose && socketConnected && socket && socket.send(
  JSON.stringify({
    handsfree: true,
    action: 'updateCursor',
    cursor: lastPose.cursor,
    face: {
      rotationX: lastPose.face.rotationX,
      rotationY: lastPose.face.rotationY,
      rotationZ: lastPose.face.rotationZ,
      // See: https://handsfree.js.org/#/docs/config
      translationX: lastPose.face.translationX,
      translationY: lastPose.face.translationY
    }
  }))
}, 400)